Second-order stochastic variational inference

Stochastic gradient descent (SGD), the workhorse of stochastic optimization, is slow in theory (sub-linear convergence) and in practice (thousands of iterations), intuitively for two reasons: 1) Its learning rate schedule is fixed a priori and decays rapidly enough to 0 that is square-summable. This learning rate schedule limits the step size and hence the rate of convergence for a Lipschitz objective function. 2) It fails to account for the objective’s curvature. Extensions to SGD like AdaGrad [7] and Adam [8] adjust for the relative scales of the parameters but not the curvature in general.